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tion, factory floor, etc.). 

(77) A portable laptop implementation can be made to deliver multimedia mail 
with video, audio and synchronized annotations via CD-ROM or an add-on 
videotape unit with separate video, audio and time code tracks (a stereo 
videotape player can use the second audio channel for time code signals). 
Videotapes or CD-ROMs can be created in main offices and express mailed, 
thus 

avoiding the need for high-bandwidth networking when on the road. Cellular 
phone links can be used to obtain both voice and data communications (via 
modems). Modem-based data communications are sufficient to support remote 
control of mail or presentation playback, annotation, file transfer and fax 
features. The laptop can then be brought into the office and attached to a 
docking station where the available MLAN 10 and additional functions adapted 
from Add-on box 800 can be supplied, providing full CMW capability. 

(78) COLLABORATIVE MULTIMEDIA WORKSTATION SOFTWARE 

(79) CMW software modules 160 are illustrated generally in FIG. 20 and 
discussed in greater detail below in conjunction with the software running on 
MLAN Server 60 of FIG. 3. Software 160 allows the user to initiate and manage 
(in conjunction with the server software) videoconferencing, data conferencing, 
multimedia mail and other collaborative sessions with other users across the 
network. 

(80) Also present on the CMW in this embodiment are standard multitasking 
operating system/GUI software 180 (e.g., Apple Macintosh System 7, Microsoft 
Windows 3.1 , or UNIX with the "X Window System" and Motif or other GUI 
"window 

manager" software) as well as other applications 170, such as word processing 
and spreadsheet programs. Software modules 161-168 communicate with 
operating 

system/GUI software 180 and other applications 170 utilizing standard function 
calls and interapplication protocols. 

(81 ) The central component of the Collaborative Multimedia Workstation 
software is the Collaboration Initiator 161. All collaborative functions can 

be accessed through this module. When the Collaboration Initiator is started, 
it exchanges initial configuration information with the Audio Video Network 
Manager (AVNM) 60 (shown in FIG. 3) through Data Network 902. Information 
is 

also sent from the Collaboration Initiator to the AVNM indicating the location 
of the user, the types of services available on that workstation (e.g., 
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(via the ADD 

button in Collaboration Initiator window 204) to provide additional information 
regarding the factual history of the security. Upon selecting the ADD button, 
video window 203 now displays, as illustrated in FIG. 38, a video mosaic 
consisting of three smaller images (instead of a single large image displaying 
only caller 231 ) of the Expert 233, caller 231 and London expert 232. 

(225) During this videoconference, an urgent PRIORITY request (New Call 
window 234) is received from the Expert's boss (who is engaged in a three-party 
videoconference call with two members of the bank's operations department and 
is attempting to add the Expert to that call to answer a quick question). The 
Expert puts his three-party videoconference on hold (merely by clicking the 
HOLD button in video window 203) and accepts (via the ACCEPT button of New 
Call 

window 234) the urgent call from his boss, which results in the Expert being 
added to the boss' three-party videoconference call. 

(226) As illustrated in FIG. 39, video window 203 is now replaced with a 
four-person video mosaic representing a four-party conference call consisting 
of the Expert 233, his boss 241 and the two members 242 and 243 of the bank's 
operations department. The Expert quickly answers the boss 1 question and, by 
clicking on the RESUME button (of video window 203) adjacent to the names of 
the other participants to the call on hold, simultaneously hangs up on the 
conference call with his boss and resumes his three-party conference call 
involving the securities issue, as illustrated in video window 203 of FIG. 40. 

(227) While that call was on hold, however, analyst 231 and London expert 232 
were still engaged in a two-way videoconference (with a blackened portion of 
the video mosaic on their screens indicating that the Expert was on hold) and 
had shared and annotated a graphical image 250 (see annotations 251 to image 
250 of FIG. 40) illustrating certain financial concerns. Once the Expert 
resumed the call, analyst 231 added the Expert to the share session, causing 
Share window 21 1 containing annotated image 250 to appear on the Expert's 
screen. Optionally, snapshot sharing could progress while the video was on 
hold. 

(228) Before concluding his conference regarding the securities, the Expert 
receives notification of an incoming multimedia mail message-e.g., a beep 
accompanied by the appearance of an envelope 252 in the dog's mouth in In 
Box 

icon 205 shown in FIG. 40. Once he concludes his call, he quickly scans his 
incoming multimedia mail message by clicking on In Box icon 205, which 
invokes 
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IG. 18A. Currently available personal computers (e.g., an Apple Macintosh or 
an IBM-compatible PC, desktop or laptop) and workstations (e.g., a Sun 
SPARCstation) can be adapted to work with the present invention to provide 
such 

features as real-time videoconferencing, data conferencing, multimedia mail, 
etc. In business situations, it can be advantageous to set up a laptop to 
operate with reduced functionality via cellular telephone links and removable 
storage media (e.g., CD-ROM, video tape with timecode support, etc.), but take 
on full capability back in the office via a docking station connected to the 
MLAN 10. This requires a voice and data modem as yet another function server 
attached to the MLAN. 

(60) The currently available personal computers and workstations serve as a 
base workstation platform. The addition of certain audio and video I/O devices 
to the standard components of the base platform 1 00 (where standard 
components 

include the display monitor 200, keyboard 300 and mouse or tablet (or other 
pointing device) 400), all of which connect with the base platform box through 
standard peripheral ports 101, 102 and 103, enables the CMW to generate and 
receive real-time audio and video signals. These devices include a video 
camera 500 for capturing the user's image, gestures and surroundings 
(particularly the user's face and upper body), a microphone 600 for capturing 
the user's spoken words (and any other sounds generated at the CMW), a 
speaker 

700 for presenting incoming audio signals (such as the spoken words of another 
participant to a videoconference or audio annotations to a document), a video 
input card 130 in the base platform 100 for capturing incoming video signals 
(e.g., the image of another participant to a videoconference, or videomail), 
and a video display card 120 for displaying video and graphical output on 
monitor 200 (where video is typically displayed in a separate window). 

(61 ) These peripheral audio and video I/O devices are readily available from 

a variety of vendors and are just beginning to become standard features in (and 
often physically integrated into the monitor and/or base platform of) certain 
personal computers and workstations. See, e.g., the aforementioned BYTE 
article ("Video Conquers the Desktop"), which describes current models of 
Apple's Macintosh AV series personal computers and Silicon Graphics' Indy 
workstations. 

(62) Add-on box 800 (shown in FIG. 18A and illustrated in greater detail in 
FIG. 19) integrates these audio and video I/O devices with additional functions 
(such as adaptive echo canceling and signal switching) and interfaces with AV 
Network 901 . AV Network 901 is the part of the MLAN 10 which carries 
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esponse thereto, the AVNM causes the A/V Switching Circuitry 30 to set up the 
appropriate connections between ports 81 and 82, as indicated by the dashed 
line 83. 

[0152] FIG. 25 diagrammatically illustrates how two-party calls are connected 
for CMWs WS-1 and WS-2 when located in different MLANs 10a and 10b. As 
illustrated in FIG. 25, CMW WS-1 of MLAN 10a is connected to a port 91a of 
A/V 

Switching Circuitry 30a of MLAN 10a, while CMW WS-2 is connected to a port 
91b 

of the AA/ Switching Circuitry 30b of MLAN 1 0b. It will be assumed that MLANs 
10a and 10b can communicate with each other via ports 92a and 92b (through 
respective WAN gateways 40a and 40b and WAN 15). A call between CMWs 
WS-1 and 

WS-2 can then be established by AVNM of MLAN 10a in response to the 
creation of 

callhandles at ports 91a and 92a, setting up appropriate connections between 
these ports as indicated by dashed line 93a, and by AVNM of MLAN 10b, in 
response to callhandles created at ports 91b and 92b, setting up appropriate 
connections between these ports as indicated by dashed line 93b. Appropriate 
paths 94a and 94b in WAN gateways 40a and 40b, respectively, are set up by 
the 

WAN network manager 65 (FIG. 21) in each network. 
Conference Calls 

[0153] Next to be described is the specific manner in which the preferred 
embodiment provides for multi-party conference calls (involving more than two 
participants) When a multi-party conference call is initiated, the CMW provides 
a screen that is similar to the screen for two-party calls, which displays a 
live video picture of the callee's image in a video window. However, for 
multi-party calls, the screen includes a video mosaic containing a live video 
picture of each of the conference participants (including the CMW user's own 
picture), as shown, for example, in FIG. 8B. Of course, other embodiments 
could show only the remote conference participants (and not the local CMW 
user) 

in the conference mosaic (or show a mosaic containing both participants in a 
two-party call). In addition to the controls shown in FIG. 8B, the multi-party 
conference screen also includes buttons/menu items that can be used to place 
individual conference participants on hold, to remove individual participants 
from the conference, to adjourn the entire conference, or to provide a 
"close-up" image of a single individual (in place of the video mosaic). 
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es. Also note that Site #2 and Site #3 are each 

required to transmit two different types of cut-and-paste mosaics. 

[0100] The preferred embodiment also provides the capability of allowing a 
conference participant to select a close-up of a participant displayed on a 
mosaic. This capability is provided whenever a full individual video picture 
is available at that user's site. In such case, the AA/ Switching Circuitry 30 
(FIG. 3) switches the selected full video picture (whether obtained locally or 
from another site) to the CMW that requests the close-up. 

[0101] Next to be described in connection with FIGS. 18A, 18B, 19 and 20 are 
various preferred embodiments of a CMW in accordance with the invention. 

Collaborative Multimedia Workstation Hardware 

[0102] One embodiment of a CMW 12 of the present invention is illustrated in 
FIG. 18A Currently available personal computers (e.g., an Apple Macintosh or 
an 

IBM-compatible PC, desktop or laptop) and workstations (e,g., a Sun SPARC 
station) can be adapted to work with the present invention to provide such 
features as real-time videoconferencing, data conferencing, multimedia mail, 
etc. In business situations, it can be advantageous to set up a laptop to 
operate with reduced functionality via cellular telephone links and removable 
storage media (e.g., CD-ROM, video tape with timecode support etc.), but take 
on full capability back in the office via a docking station connected to the 
MLAN 10. This requires a voice and data modem as yet another function server 
attached to the MLAN. 

[0103] The currently available personal computers and workstations serve as a 
base workstation platform. The addition of certain audio and video I/O devices 
to the standard components of the base platform 100 (where standard 
components 

include the display monitor 200, keyboard 300 and mouse or tablet (or other 
pointing device) 400), all of which connect with the base platform box through 
standard peripheral ports 101, 102 and 103, enables the CMW to generate and 
receive real-time audio and video signals. These devices include a video 
camera 500 for capturing the user's image, gestures and surroundings 
(particularly the user's face and upper body), a microphone 600 for capturing 
the user's spoken words (and any other sounds generated at the CMW), a 
speaker 

700 for presenting incoming audio signals (such as the spoken words of another 
participant to a videoconference or audio annotations to a document), a video 
input card 130 
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Side Mount 850 eliminates 

the necessity of external connections to these integrated audio and video I/O 
devices, and includes an LCD display 810 for displaying the incoming video 
signal (which thus eliminates the need for a base platform video input card 
130). 

[01 19] Given the proximity of Side Mount device 850 to the user, and the direct 
access to audio/video I/O within that device, various additional controls 820 
can be provided at the user's touch (all well within the capabilities of those 
skilled in the art). Note that, with enough additions, Side Mount unit 850 can 
become virtually a standalone device that does not require a separate computer 
for services using only audio and video. This also provides a way of 
supplementing a network of full-feature workstations with a few low-cost 
additional "audio video intercoms" for certain sectors of an enterprise (such 
as clerical, reception, factory floor, etc.). 

[0120] A portable laptop implementation can be made to deliver multimedia mail 
with video, audio and synchronized annotations via CD-ROM or an add-on 
videotape unit with separate video, audio and time code tracks (a stereo 
videotape player can use the second audio channel for time code signals). 
Videotapes or CD-ROMs can be created in main offices and express mailed, 
thus 

avoiding the need for high-bandwidth networking when on the road. Cellular 
phone links can be used to obtain both voice and data communications (via 
modems) Modem-based data communications are sufficient to support remote 
control of mail or presentation playback, annotation, file transfer and fax 
features. The laptop can then be brought into the office and attached to a 
docking station where the available MLAN 10 and additional functions adapted 
from Add-on box 800 can be supplied, providing full CMW capability. 

Collaborative Multimedia Workstation Software 

[0121] CMW software modules 160 are illustrated generally in FIG. 20 and 
discussed in greater detail below in conjunction with the software running on 
MLAN Server 60 of FIG. 3 Software 160 allows the user to initiate and manage 
(in conjunction with the server software) videoconferencing, data conferencing, 
multimedia mail and other collaborative sessions with other users across the 
network. 

[0122] Also present on the CMW in this embodiment are standard multitasking 
operating system/GUI software 180 (e.g., Apple Macintosh System 7, Microsoft 
Windows 3. 1 , or UNIX with the "X Window System" and Motif or other GUI 
"window 
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ine 93a t and by AVNM of 

MLAN 10b t in response to callhandles created at ports 91b and 92b, setting up 
appropriate connections between these ports as indicated by dashed line 93b. 
Appropriate paths 94a and 94b in WAN gateways 40a and 40b, respectively are 
set 

up by the WAN network manager 65 (FIG. 21) in each network. 

(115) CONFERENCE CALLS 

(116) Next to be described is the specific manner in which the preferred 
embodiment provides for multi-party conference calls (involving more than two 
participants). When a multi-party conference call is initiated, the CMW 
provides a screen that is similar to the screen for two-party calls, which 
displays a live video picture of the callee's image in a video window. 
However, for multi-party calls, the screen includes a video mosaic containing a 
live video picture of each of the conference participants (including the CMW 
user's own picture), as shown, for example, in FIG. 8B. Of course, other 
embodiments could show only the remote conference participants (and not the 
local CMW user) in the conference mosaic (or show a mosaic containing both 
participants in a two-party call). In addition to the controls shown in FIG. 

8B, the multi-party conference screen also includes buttons/menu items that can 
be used to place individual conference participants on hold, to remove 
individual participants form the conference, to adjourn the entire conference, 
or to provide a "close-up" image of a single individual (in place of the video 
mosaic). 

(117) Multi-party conferencing requires all the mechanisms employed for 
2-party calls. In addition, it requires the conference bridge manager CBM 64 
(FIG. 21 ) and the conference bridges 36 (FIG. 3). The CBM acts as a client of 
the AVNM in managing the operation of the conference bridges 36. The CBM 
also 

acts a server to other clients on the network. The CBM makes conferencing 
services available by creating service records of type "conference" in the AVNM 
service database and associating these services with the ports on AN Switching 
Circuitry 30 for connection to conference bridges 36. 

(1 1 8) The preferred embodiment provides two ways for initiating a conference 
call. The first way is to add one or more parties to an existing two-party 

call. For this purpose, an ADD button is provided by both the Collaboration 
Initiator and the Rolodex, as illustrated in FIGS. 2A and 22. To add a new 
party, a user selects the party to be added (by clicking on the user's rolodex 
name or face icon as described above) and clicks on the ADD button to invite 
that new party. Additional parties can be invited in a similar manner. The 
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